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based learning resource, which was used to augment the delivery of a second 
year degree level module. The development of the learning resource was 
informed by the educational theories expounded in the conversational 
framework, Laurillard (1993) . Subsequently, the resource was evaluated using 
criteria extracted from the conversational framework as a set of guidelines. 
It was found that all of the criteria could be supported to some extent, 
although in some areas the support was marginal. The learning resource was 
then evaluated by students who were asked to give details about which of the 
individual facilities were used, and beyond this, which were useful. The 
results of this student centered evaluation showed that the students did not 
consider several key facilities "useful." It is suggested that further work 
be done on developing a method for determining criteria, implementing a 
resource and evaluating the facilities for successfulness, as determined by 
students. (Author) 
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Abstract: This paper details a study into the development of a WebCT based 
learning resource, which was used to augment the delivery of a second year degree 
level module. The development of the learning resource was informed by the 
educational theories expounded in the conversational framework, Laurillard (1993). 
Subsequently the resource was evaluated using criteria extracted from the 
conversational framework as a set of guidelines. It was found that all of the criteria 
could be supported to some extent, although in some areas the support was 
marginal. The learning resource was then evaluated by students who were asked to 
give details about which of the individual facilities were used, and beyond this 
which were useful. The results of this student centered evaluation showed that the 
students did not consider several key facilities ’useful’. It is suggested that further 
work be done on developing a method for determining criteria, implementing a 
resource and evaluating the facilities for successfulness, as determined by students. 



Introduction 

The process of defining educational criteria with which to inform the development of a computer based 
learning resource is well known. There are a wide range of criteria, informed for the most part by the 
theoretical assertions of a variety of pedagogic traditions and paradigms. Most of us involved in 
developing computer based learning resources will be familiar to some extent with one or more of these 
traditions. The process of choosing criteria, developing the resource and analysing the extent to which 
the attributes in the learning resource support the criteria is also well known. In addition to this, many 
of us are familiar with evaluative measures that seek to shed light on the successfulness of the resultant 
learning resource in general terms. Often these take the form of questionnaires seeking to elicit student 
attitudes to the learning resource as a whole. Thus we have on the one hand pedagogic objectives, and 
on the other hand student evaluation as to the success overall of the learning resource. However these 
two elements are not necessarily directly connected. It is possible to have met a set of criteria in 
creating a successfully learning resource, without those criteria having had significant bearing upon the 
outcome. 

This paper is based upon a case study learning resource, constructed using WebCT. 

The design of the learning resources was informed making use of criteria taken from the conversational 
Framework devised by Laurillard (1993). The case study is described in the next section, followed by 
an explanation of the criteria taken from the conversational framework. 

The case study 

The case study module under consideration is IC240 module, ’IT in business’, a second year module 
provided by the division of Information and Communication Technologies, which is part of the 
University of Abertay Dundee. The students of IC240 are studying for degrees in accountancy, 
tourism, business, management and a range of other courses. This diverse mixture of student 
backgrounds means that there is a wide range in levels of previous experience in IT. In practical terms 
this diversity of student experience suggests that a flexible learning resource would be beneficial to 
students. For this reason a learning resource making use of the Virtual Learning Environment (VLE) 
WebCT was developed. 
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WebCT is an integrated web based learning environment. It was devised at the university of British 
Columbia, www.webct.com . The software is based upon web technology, offering a number of 
facilities running as Java scripts or applets. In this study the vast majority of WebCT facilities 
available were deployed, Table 1 outlines the facilities and how they were used. 



Table 1 WebCT facilities/use 

Navigation model 

Home page with 

icons/hyperlinks to resources. 

Content 

Module Material 

Quiz 

Formative and summative 

assessments 

Survey 

Evaluative feedback 
Goals 

Learning objectives 



Discussion Forum 

Course questions 
My progress 
Charting access 

Chat 

Questions to tutor and other 
students participation 

Whiteboard 

Student interaction Student 

tracking 

Register 



Compile 

Sections of module material 

Calendar 

Module information and 

deadlines 

References 

Links to external resources 

Student Web Pages & 

presentations 

Student created content 

Glossary 

Defined terms 



The case study learning resource was the subject of a two-fold evaluation. Firstly the learning resource 
was evaluated heuristically against the criteria extracted from the conversational framework. 
Subsequently the individual facilities comprising the learning resource were tested by means of a 
survey of student opinion. 



The Conversational Framework 

The heuristic evaluation made use of the Conversational Framework developed by Laurillard (1993) as 
an evaluation tool for WebCT. This framework was chosen as the work encapsulated in the framework 
has pioneered the evaluation of hypermedia based learning resources. Authors such as Brittain (1999) 
have argued that whether the VLE is being used for distance learning or to enhance learning within 
institutions, their most important role is as a medium for supporting constructivist and conversational 
approaches to learning (Brittain 1999). Therefore Laurillard’s conversational model offers itself as an 
interesting candidate for providing the basis of a heuristic pedagogical evaluation framework for 
WebCT. 

The roots of the framework lie in the Conversation Theory developed by Gordon Pask (1976), although 
Laurillard traces the need for dialogue in the learning process back to the Socratic method of 
philosophical enquiry. The centrality of dialogue in the model comes from the need for the teacher to 
unearth the student’s mental constructs about a topic before negotiating the path to the target conception 
that is the goal of learning from the teacher’s perspective. 

The teaching strategy advocated in the model is based on interaction between lecturer and student and 
not solely on the actions required of the student. The model advocates that action on the part of the 
student is constructed around the dialogue and should be supplemented by constructive and meaningful 
feedback from the teacher. There are a number of key characteristics of the conversational model as 
applied to academic learning. These are drawn from Laurillard, (1993, pp.94-95). 

1 . T(eacher) can describe conception 

2. S(tudent) can describe conception 

3. T can redescribe in the light of S's conception or action 

4. S can redescribe in light of T's redescription or action 

5. T can adapt task goal in light of S’s description or action. 

6. T can set task goal 

7. S can act to achieve task goal 

8. T can set up world to give intrinsic feedback on actions 

9. S can modify in light of feedback on action 

10. S can adapt actions in light of T’s description 

11. S can reflect on interaction to modify redescription 

1 2. T can reflect on S's action to modify redescription 
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It is important to note that these criteria are not necessarily sequential, but it is suggested that they 
should be supported by WebCT if it is to be judged compliant with this educational model. In the next 
section we will explore the extent to which the case study module is successful in delivering the 
criteria, but before this we will examine an important new revision of the conversational framework. 

Conversational Framework revision 1999 

The first version of the conversational framework (Laurillard, 1993) described the learning process and 
was used as an analytic tool for information technology media and methods. However, the original 
framework did not indicate a process whereby the student internalised their learning, rather it focuses 
entirely upon a dialogue between student and teacher. This omission has been dealt with in an article 
'A conversational framework for individual learning applied to the 'learning organisation' and the 
learning society.' (Laurillard, 1999). Clearly this revision addresses a crucial part of learning, and as 
such it is suggested that the evaluative criteria be expanded to encompass this element. Therefore for 
the purposes of this evaluation the full set of criteria will be numbers 1-12 as indicated above and in 
addition; 

13. S can internalise articulati on/action. 

The Heuristic Evaluation 

A suitable evaluation framework for WebCT using the conversational model could be constructed in a 
variety of ways. The way suggested in this study involved constructing a table that describes the tools 
that are available for each of the stages of interaction described in the model (see Table 2). Table 2 
shows the evaluation framework for WebCT using the interactions in the conversational model as 
criteria against which to identify the tools provided by WebCT. For each element a WebCT tool is 
identified along with a description of how it was used in the case study module. 

This relatively simple approach offers much potential as a methodology for evaluating virtual learning 
environments (VLE) such as WebCT. A similar proposal has been recently put forward by Crawley, 
(1999). However, the major emphasis in this study is the way the evaluation by students, given in the 
next section, is used to validate the findings put forward by the theoretical evaluation. 

In order to evaluate WebCT using the conversational framework we need to establish which tools are 
provided within the software to allow dialogue and action to mutually influence each other to allow 
modification of both conceptions and actions on the part of the student as described above. Another 
issue that quickly becomes apparent is that the notion of a 'micro-world', takes on a different meaning 
in the case of WebCT than more traditional forms of courseware, Britain (1999). In essence, WebCT 
provides the tools for a lecturer to build a micro-world by allowing the teacher to construct learning 
activities enriched by the resources. 



Table 2 Conversational framework evaluation of WebCT 



CHARACTERISTIC 


SUPPORTED 


IC240 example 


1. T can describe conception 


Goals, Module content. Ref., Calendar 


Database exercise, 
Theoretical content. 


2. S can describe conception 


Discussion forum, Chat, Whiteboard, 
Web pages 


Questions, 
Remote lab 


3. T can redescribe in the light of S's 
conception or action 


Discussion forum, Chat, Whiteboard. 


Answers, 
Remote lab 


4. S can redescribe in light of T's 
redescription or action 


Discussion forum, Chat, Whiteboard 
Web pages 


Questions, 
Remote lab 


5. T can adapt task goal in light of S's 
description or action. 


Goals, Module content, References. 


Discussion 


6. T can set task goal 


Goals, Module content. 


Learning objectives 

Database exercises, quiz 
information. 


7. S can act to achieve task goal 


Quiz, Compile 


Build notes 
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Review test, 
Examination. 


8. T can set up world to give intrinsic 
feedback on actions 


Quiz, Progress 


Review test. 


9. S can modify in light of feedback 
on action 


Quiz 


Review test. 


10. S can adapt actions in light of T's 
description 


Quiz 


Review test. 


1 1 . S can reflect on interaction to 
modify redescription 


Discussion forum, Chat, Whiteboard. 


Discussions 


12. T can reflect on S's action to modify 
redescription. 


Discussion forum, Chat, Whiteboard. 


Discussions 


13. S can internalise articulation/action. 


Note editor. 


Independent study. 



The points described in the table show that WebCT as used for the module in this study has, at least to 
some extent, the facilities to support all of the features laid out by the conversational model. However 
it is important to consider how successful the learning resource is at delivering the requirements 
specified in the conversational framework. It is the results of this evaluation, given in the next section, 
that will be used to make a judgement as to whether this particular resource is successfully compliant 
with the criteria in this educational evaluation. 

Student centered evaluation 

In this study the evaluative measures included an 'exit survey'. The rationale for the exit survey was to 
find out directly what the experience of using WebCT was like for the students. The students were 
asked about their experience in general, where it was discovered that 13 of the 15 students taking part 
in the evaluation, found the resource as a whole useful, 2 were neutral, none negative. It is important to 
note that only fifteen students were involved in the survey, representing around a quarter of the total 
number of student studying the module. These numbers mean that the findings can only be regarded as 
indicative, carried out within the context of a pilot study. The next two questions in the survey sought 
to shed light into which facilities the students used, and beyond this which they found useful. This 
distinction between used and useful is crucial for this study. It is suggested that it is not sufficient to 
say that a theory indicates the use of a facility, and consequently its inclusion validates the learning 
resource. If students do not find the facilities useful it throws into doubt the voracity of using the 
facility to achieve a particular educational objective. The questions and findings where as follows: 

> From the list of WebCT facilities which have you USED? 

> From the list of WebCT facilities which did you find USEFUL? 



Table 3 The results of the student evaluation ’exit survey* 



Response 


Number of 
response 
(used) 


Number of 
responses 
(useful) 


Percentage 
of total 
Useful 


1 . Module content 


15 


15 


100% 


2. Discussion Forum 


9 


7 


46% 


3. Chat 


9 


5 


33% 


4. Calendar 


15 


12 


80% 


5. Quiz Review tests 


15 


15 


100% 


6. Whiteboard 


7 


1 


7% 


7. Check up progress 


4 


0 


0% 


8. Compile notes 


9 


6 


40% 


9.Targets/goals 


4 


3 


26% 


10. References 


8 


8 


53% 


1 1 . Web page authoring 


9 


6 


40% 


12. Note editor 


4 


1 


7% 
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These results indicate that some facilities were used but not found to be useful, in particular the note 
editor, discussion forum, chat, whiteboard, check progress and targets were not useful to all of the 
students who used them. In addition, some of the students elected to not use some of the facilities. As 
some of these facilities were key to supporting the conversational framework, this finding poses 
searching questions about the context for their use. Further analyses of the findings regarding the 
usefulness of the facilities are shown in table 4 on the next page. Table 4 shows the numbered items 
from the educational criteria along the top (listed in table 5), whilst the WebCT facilities are listed 
down the left hand side. This form of analysis i.e. the conversational framework analysed against the 
facilities in a VLE was used by Conole (1998). It is suggested however, that this analysis goes a step 
further, in so far as it makes use of student evaluation to inform the rate of successfulness. The table 
highlights the points discussed earlier, for example some of the requirements are not being successfully 
met. 

Table 4, on the next page, shows that as the module content display was useful to 100% of students, 
therefore criteria 1 , 5 and 6 of the conversational framework were successfully implemented. On the 
other hand, the note editor facility was useful to only 7% of students, therefore criteria 13 can not be 
said to be successfully implemented. Reading the table starting from the criteria, we can see criteria 6 
Teacher set task goal, is successfully implemented by the module content facility if not entirely by the 
goal setting facility. Perhaps most significantly for the conversational framework the facilities being 
used to support interaction between Student and Teacher, for example criteria 2,3 and 4, are not being 
entirely successfully implemented by facilities such as the discussion forum. 



Table 5 Conversational framework Criteria 








1. 


T(eacher) can describe conception 




8. 


T can set up world to give intrinsic 


2. 


S(tudent) can describe conception 






feedback on actions 


3. 


T can re-describe in the light of 


S’s 


9. 


S can modify in light of feedback on action 




conception or action 




10. 


S can adapt actions in light of T's 


4. 


S can re-describe in light of T's 


re- 




description 




description or action 




11. 


S can reflect on interaction to modify re- 


5. 


T can adapt task goal in light of 


S's 




description 




description or action. 




12. 


T can reflect on S's action to modify re- 


6. 


T can set task goal 






description 


7. 


S can act to achieve task goal 




13. 


S can internalize articulation/action 



Conclusion 

In conclusion it is suggested that whilst the WebCT software supports the design and development of a 
learning resource that can be compliant with an educational evaluation based upon the conversational 
framework. It can not, however, be taken for granted that any individual learning resource is entirely 
successful in delivering the aims of the conversational framework. The results of this study indicates 
that success maybe illusive, even in terms of the fundamental requirements. In respect of the IC240 
module, it is suggested that the learning resource be redeveloped, in order to leverage more 
successfully key facilities in the learning resources. In more general terms, it is suggested that the 
process of developing pedagogically informed learning resources, going from criteria, to facilities and 
then to evaluation, should be refined into a methodology that focuses upon the student as the main 
arbiter of successfulness. 
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Table 4 Successfulness! Conversational framework criteria vs student evaluation. 
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